{
 "cells": [
  {
   "cell_type": "raw",
   "id": "e9437c8a-d8b7-4bf6-8ff4-54068a5a266c",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 1.5\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0df7646-b1e1-4014-a841-6dae9b3c50d9",
   "metadata": {},
   "source": [
    "# How to stream chat model responses\n",
    "\n",
    "All [chat models](https://api.js.langchain.com/classes/langchain_core.language_models_chat_models.BaseChatModel.html) implement the [Runnable interface](https://.api.js.langchain.com/classes/langchain_core.runnables.Runnable.html), which comes with **default** implementations of standard runnable methods (i.e. `invoke`, `batch`, `stream`, `streamEvents`). This guide covers how to use these methods to stream output from chat models.\n",
    "\n",
    ":::{.callout-tip}\n",
    "\n",
    "The **default** implementation does **not** provide support for token-by-token streaming, and will instead return an [`AsyncGenerator`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/AsyncGenerator) that will yield all model output in a single chunk. It exists to ensures that the the model can be swapped in for any other model as it supports the same standard interface.\n",
    "\n",
    "The ability to stream the output token-by-token depends on whether the provider has implemented token-by-token streaming support.\n",
    "\n",
    "You can see which [integrations support token-by-token streaming here](/docs/integrations/chat/).\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a76660e-7691-48b7-a2b4-2ccdff7875c3",
   "metadata": {},
   "source": [
    "## Streaming\n",
    "\n",
    "Below, we use a `---` to help visualize the delimiter between tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c84c144d",
   "metadata": {},
   "source": [
    "```{=mdx}\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs />\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2331cfb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "// @lc-docs-hide-cell\n",
    "import { ChatAnthropic } from \"@langchain/anthropic\";\n",
    "\n",
    "const model = new ChatAnthropic({\n",
    "  model: \"claude-3-5-sonnet-20240620\",\n",
    "});"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "975c4f32-21f6-4a71-9091-f87b56347c33",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "---\n",
      "Here's\n",
      "---\n",
      " a one\n",
      "---\n",
      "-\n",
      "---\n",
      "verse song about goldfish on\n",
      "---\n",
      " the moon:\n",
      "\n",
      "Verse\n",
      "---\n",
      ":\n",
      "Swimming\n",
      "---\n",
      " through the stars\n",
      "---\n",
      ",\n",
      "---\n",
      " in\n",
      "---\n",
      " a cosmic\n",
      "---\n",
      " lag\n",
      "---\n",
      "oon\n",
      "---\n",
      "\n",
      "Little\n",
      "---\n",
      " golden\n",
      "---\n",
      " scales\n",
      "---\n",
      ",\n",
      "---\n",
      " reflecting the moon\n",
      "---\n",
      "\n",
      "No\n",
      "---\n",
      " gravity to\n",
      "---\n",
      " hold them,\n",
      "---\n",
      " they\n",
      "---\n",
      " float with\n",
      "---\n",
      " glee\n",
      "Goldfish\n",
      "---\n",
      " astron\n",
      "---\n",
      "auts, on a lunar\n",
      "---\n",
      " sp\n",
      "---\n",
      "ree\n",
      "---\n",
      "\n",
      "Bub\n",
      "---\n",
      "bles rise\n",
      "---\n",
      " like\n",
      "---\n",
      " com\n",
      "---\n",
      "ets, in the\n",
      "---\n",
      " star\n",
      "---\n",
      "ry night\n",
      "---\n",
      "\n",
      "Their fins like\n",
      "---\n",
      " tiny\n",
      "---\n",
      " rockets, a\n",
      "---\n",
      " w\n",
      "---\n",
      "ondrous sight\n",
      "Who\n",
      "---\n",
      " knew\n",
      "---\n",
      " these\n",
      "---\n",
      " small\n",
      "---\n",
      " creatures\n",
      "---\n",
      ",\n",
      "---\n",
      " could con\n",
      "---\n",
      "quer space?\n",
      "---\n",
      "\n",
      "Goldfish on the moon,\n",
      "---\n",
      " with\n",
      "---\n",
      " such\n",
      "---\n",
      " fis\n",
      "---\n",
      "hy grace\n",
      "---\n",
      "\n",
      "---\n",
      "\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "const stream = await model.stream(\"Write me a 1 verse song about goldfish on the moon\")\n",
    "\n",
    "for await (const chunk of stream) {\n",
    "  console.log(`${chunk.content}\\n---`);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c61e1309-3b6e-42fb-820a-2e4e3e6bc074",
   "metadata": {},
   "source": [
    "## Stream events\n",
    "\n",
    "Chat models also support the standard [`streamEvents()`](https://api.js.langchain.com/classes/langchain_core.runnables.Runnable.html#streamEvents) method to stream more granular events from within chains.\n",
    "\n",
    "This method is useful if you're streaming output from a larger LLM application that contains multiple steps (e.g., a chain composed of a prompt, chat model and parser):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "27bd1dfd-8ae2-49d6-b526-97180c81b5f4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\n",
       "  {\n",
       "    event: \u001b[32m\"on_chat_model_start\"\u001b[39m,\n",
       "    data: { input: \u001b[32m\"Write me a 1 verse song about goldfish on the moon\"\u001b[39m },\n",
       "    name: \u001b[32m\"ChatAnthropic\"\u001b[39m,\n",
       "    tags: [],\n",
       "    run_id: \u001b[32m\"d60a87d6-acf0-4ae1-bf27-e570aa101960\"\u001b[39m,\n",
       "    metadata: {\n",
       "      ls_provider: \u001b[32m\"openai\"\u001b[39m,\n",
       "      ls_model_name: \u001b[32m\"claude-3-5-sonnet-20240620\"\u001b[39m,\n",
       "      ls_model_type: \u001b[32m\"chat\"\u001b[39m,\n",
       "      ls_temperature: \u001b[33m1\u001b[39m,\n",
       "      ls_max_tokens: \u001b[33m2048\u001b[39m,\n",
       "      ls_stop: \u001b[90mundefined\u001b[39m\n",
       "    }\n",
       "  },\n",
       "  {\n",
       "    event: \u001b[32m\"on_chat_model_stream\"\u001b[39m,\n",
       "    run_id: \u001b[32m\"d60a87d6-acf0-4ae1-bf27-e570aa101960\"\u001b[39m,\n",
       "    name: \u001b[32m\"ChatAnthropic\"\u001b[39m,\n",
       "    tags: [],\n",
       "    metadata: {\n",
       "      ls_provider: \u001b[32m\"openai\"\u001b[39m,\n",
       "      ls_model_name: \u001b[32m\"claude-3-5-sonnet-20240620\"\u001b[39m,\n",
       "      ls_model_type: \u001b[32m\"chat\"\u001b[39m,\n",
       "      ls_temperature: \u001b[33m1\u001b[39m,\n",
       "      ls_max_tokens: \u001b[33m2048\u001b[39m,\n",
       "      ls_stop: \u001b[90mundefined\u001b[39m\n",
       "    },\n",
       "    data: {\n",
       "      chunk: AIMessageChunk {\n",
       "        lc_serializable: \u001b[33mtrue\u001b[39m,\n",
       "        lc_kwargs: {\n",
       "          content: \u001b[32m\"\"\u001b[39m,\n",
       "          additional_kwargs: \u001b[36m[Object]\u001b[39m,\n",
       "          tool_calls: [],\n",
       "          invalid_tool_calls: [],\n",
       "          tool_call_chunks: [],\n",
       "          response_metadata: {}\n",
       "        },\n",
       "        lc_namespace: [ \u001b[32m\"langchain_core\"\u001b[39m, \u001b[32m\"messages\"\u001b[39m ],\n",
       "        content: \u001b[32m\"\"\u001b[39m,\n",
       "        name: \u001b[90mundefined\u001b[39m,\n",
       "        additional_kwargs: {\n",
       "          id: \u001b[32m\"msg_01JaaH9ZUXg7bUnxzktypRak\"\u001b[39m,\n",
       "          type: \u001b[32m\"message\"\u001b[39m,\n",
       "          role: \u001b[32m\"assistant\"\u001b[39m,\n",
       "          model: \u001b[32m\"claude-3-5-sonnet-20240620\"\u001b[39m\n",
       "        },\n",
       "        response_metadata: {},\n",
       "        id: \u001b[90mundefined\u001b[39m,\n",
       "        tool_calls: [],\n",
       "        invalid_tool_calls: [],\n",
       "        tool_call_chunks: [],\n",
       "        usage_metadata: \u001b[90mundefined\u001b[39m\n",
       "      }\n",
       "    }\n",
       "  },\n",
       "  {\n",
       "    event: \u001b[32m\"on_chat_model_stream\"\u001b[39m,\n",
       "    run_id: \u001b[32m\"d60a87d6-acf0-4ae1-bf27-e570aa101960\"\u001b[39m,\n",
       "    name: \u001b[32m\"ChatAnthropic\"\u001b[39m,\n",
       "    tags: [],\n",
       "    metadata: {\n",
       "      ls_provider: \u001b[32m\"openai\"\u001b[39m,\n",
       "      ls_model_name: \u001b[32m\"claude-3-5-sonnet-20240620\"\u001b[39m,\n",
       "      ls_model_type: \u001b[32m\"chat\"\u001b[39m,\n",
       "      ls_temperature: \u001b[33m1\u001b[39m,\n",
       "      ls_max_tokens: \u001b[33m2048\u001b[39m,\n",
       "      ls_stop: \u001b[90mundefined\u001b[39m\n",
       "    },\n",
       "    data: {\n",
       "      chunk: AIMessageChunk {\n",
       "        lc_serializable: \u001b[33mtrue\u001b[39m,\n",
       "        lc_kwargs: {\n",
       "          content: \u001b[32m\"Here's\"\u001b[39m,\n",
       "          additional_kwargs: {},\n",
       "          tool_calls: [],\n",
       "          invalid_tool_calls: [],\n",
       "          tool_call_chunks: [],\n",
       "          response_metadata: {}\n",
       "        },\n",
       "        lc_namespace: [ \u001b[32m\"langchain_core\"\u001b[39m, \u001b[32m\"messages\"\u001b[39m ],\n",
       "        content: \u001b[32m\"Here's\"\u001b[39m,\n",
       "        name: \u001b[90mundefined\u001b[39m,\n",
       "        additional_kwargs: {},\n",
       "        response_metadata: {},\n",
       "        id: \u001b[90mundefined\u001b[39m,\n",
       "        tool_calls: [],\n",
       "        invalid_tool_calls: [],\n",
       "        tool_call_chunks: [],\n",
       "        usage_metadata: \u001b[90mundefined\u001b[39m\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "const eventStream = await model.streamEvents(\n",
    "  \"Write me a 1 verse song about goldfish on the moon\",\n",
    "  {\n",
    "    version: \"v2\"\n",
    "  },\n",
    ");\n",
    "\n",
    "const events = [];\n",
    "for await (const event of eventStream) {\n",
    "  events.push(event);\n",
    "}\n",
    "\n",
    "events.slice(0, 3);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abacb301",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "You've now seen a few ways you can stream chat model responses.\n",
    "\n",
    "Next, check out this guide for more on [streaming with other LangChain modules](/docs/how_to/streaming)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Deno",
   "language": "typescript",
   "name": "deno"
  },
  "language_info": {
   "file_extension": ".ts",
   "mimetype": "text/x.typescript",
   "name": "typescript",
   "nb_converter": "script",
   "pygments_lexer": "typescript",
   "version": "5.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
